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DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON,  VIRGINIA  22202-2884 


June  2,  1994 

MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  NAVY  (FINANCIAL 

MANAGEMENT) 

DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 

SUBJECT:  Milestone  Review  Process  for  the  Consolidated  Automated  Support  System 
(Report  No.  94-116) 


We  are  providing  this  report  for  your  review  and  comments.  This  report  is  the 
first  in  a  series  of  reports  resulting  from  our  audit  of  the  milestone  review  process  for 
Component-managed  acquisition  programs.  We  requested  comments  on  a  draft  of  this 
report.  Formal  comments  were  not  received;  however,  the  report  incorporates 
informal  comments  and  feedback  related  to  the  March  1994  Navy  Program  Decision 
Meeting. 

As  required  under  DoD  Directive  7650.3,  "Followup  on  General  Accounting 
Office,  DoD  Inspector  General,  and  Internal  Audit  Reports,"  September  5,  1989,  all 
audit  recommendations  must  be  resolved  promptly.  Therefore,  you  must  provide  final 
comments  on  the  recommendations  by  August  1,  1994.  The  recommendations  are 
subject  to  resolution  in  accordance  with  DoD  Directive  7650.3  in  the  event  of 
nonconcurrence  or  failure  to  comment.  We  also  ask  that  your  comments  indicate  ^ 
concurrence  or  nonconcurrence  with  the  material  internal  control  weakness  highlighted 
in  Parti. 


We  appreciate  the  courtesies  extended  to  the  audit  staff.  If  you  have  any 
questions  on  this  report,  please  contact  Mr.  Jack  D.  Snider,  Project  Manager,  at 
(703)  693-0402  (DSN  223-0402).  Appendix  E  lists  the  distribution  of  this  report. 
Audit  team  members  are  listed  inside  back  cover. 


Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  94-116  June  2, 1994 

(Project  No.  3 AE -*0061. 00) 

MILESTONE  REVIEW  PROCESS  FOR  THE  CONSOLIDATED 
AUTOMATED  SUPPORT  SYSTEM 


EXECUTIVE  SUMMARY 


Introduction.  The  Navy's  Consolidated  Automated  Support  System  (CASS)  includes 
standardized  automatic  test  equipment  with  computer-assisted,  multi-function 
capabilities  with  potential  multi-Service  application  to  support  testing  of  aircraft 
subsystems  and  missiles.  The  Navy  Acquisition  Executive  approved  low-rate  initial 
production  for  55,  60,  and  68  CASS  stations,  in  September  1990,  July  1992,  and  June 
1993,  respectively.  A  Navy  program  decision  meeting  for  a  Milestone  III,  Production 
Approval,  decision  was  scheduled  for  December  1993;  however,  the  decision  was 
deferred  until  March  1994.  On  March  25,  the  program  decision  meeting  was  held, 
approving  entry  into  the  production  and  deployment  phase  of  the  acquisition  process. 

Objective.  The  audit  objective  was  to  evaluate  the  effectiveness  of  the  milestone 
review  process  for  Component-managed  acquisition  programs.  The  audit  also  assessed 
the  adequacy  of  the  information  provided  to  DoD  Component  milestone  decision 
authorities  in  support  of  major  milestone  and  program  reviews  and  evaluated  internal 
controls  related  to  the  objective.  The  CASS  Program  was  one  program  reviewed  in  our 
ongoing  audit  of  the  Milestone  Review  Process  for  Component-Managed  Acquisition 
Programs. 

Audit  Results.  The  overall  CASS  Program  was  not  ready  to  proceed  into  the 
production  and  deployment  phase  of  the  acquisition  process.  Successful  completion  of 
additional  operational  test  and  evaluation  and  verified  correction  of  testing  and 
configuration  audit  deficiencies  were  required.  As  a  result,  the  CASS  Program  could 
potentially  have  passed  a  key  acquisition  milestone  without  plans  for  adequate  closure 
of  action  items  or  'correction  of  deficiencies.  The  Navy  initiated  timely  corrective 
action  on  the  deficiencies  identified  in  this  audit  concerning  the  CASS  Program  and 
deferred  the  Navy  program  decision  meeting  until  March  1994.  The  corrective  action 
should  be  completed  before  the  FY  1995  production  buy. 

Internal  Controls.  The  audit  identified  a  material  internal  control  weakness  in  that 
Navy  policy  on  performance  of  cost  and  operational  effectiveness  analyses  conflicted 
with  DoD  acquisition  regulations.  This  internal  control  weakness  is  summarized  in 
Part  I  and  fully  discussed  in  Part  n  of  this  report. 

Potential  Benefits  of  Audit.  Potential  monetary  benefits  are  not  quantifiable. 
Implementation  of  the  recommendations  will  ensure  that  decisionmakers  have  all 
available  information  to  make  fully  informed  decisions  concerning  whether  the  CASS 
Program  is  ready  to  proceed  into  production  and  whether  proposed  program  plans  for 
the  subsequent  acquisition  stage  is  consistent  with  sound  acquisition  management 
practices  (Appendix  C). 


*A  CASS  station  is  a  collection  of  multi-function  automatic  test  equipment  structured  around  a  common 
core  with  four  testing  configurations. 


Summary  of  Recommendations.  We  recommended  conducting  a  Milestone  HI 
review  of  the  CASS  Program  limited  to  the  Lot  IV  and  V  buys  and  the  configuration 
units  that  have  completed  testing,  deferring  the  award  of  the  Lot  VI  buy  until 
successful  completion  of  additional  operational  test  and  evaluation  and  a  Navy  program 
decision  meeting  on  the  overall  program,  completing  the  Operational  Testing-m  phase 
as  part  of  the  evaluation  of  operational  effectiveness  and  suitability  of  the  CASS 
Program,  correcting  the  deficiencies  identified  during  functional  and  physical 
configuration  audits  of  the  CASS  Program  before  the  Lot  V  buy,  reviewing  CASS 
Program  cost  and  benefit  analyses  to  determine  whether  the  documents  contain  the 
elements  of  a  cost  and  operational  effectiveness  analysis  needed  to  support  the 
Milestone  III  decision,  and  issuing  policy  requiring  that  cost  and  operational 
effectiveness  analyses  be  conducted  for  future  Milestone  in  decisions.  We  also 
recommended  that  the  Director,  Operational  Test  and  Evaluation,  designate  the  CASS 
Program  for  oversight  and  decide  on  the  operational  effectiveness  and  suitability  of  the 
CASS  Program. 

Management  Comments.  We  requested  comments  on  a  draft  of  this  report  from  the 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  and  the 
Director,  Operational  Test  and  Evaluation;  however,  comments  were  not  received.  As 
a  result,  we  require  comments  on  this  final  report  by  August  1,  1994. 
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The  Acquisition  Management  Directorate,  Office  of  the  Assistant  Inspector  General  for 
Auditing,  DoD,  prepared  this  report. 


Part  I  -  Introduction 


Introduction 


Background 


This  report  discusses  the  adequacy  of  the  information,  including  operational  test 
and  evaluation,  provided  to  die  Navy  milestone  decision  authorities  in  support 
of  a  Navy  program  decision  meeting  (NPDM)  for  a  Milestone  HI,  Production 
Approval,  decision  for  the  Navy's  Consolidated  Automated  Support  System 
(CASS)  Program. 

Navy  Program  Decision  Meeting.  The  NPDM  is  the  Navy's  forum  for 
acquisition  program  milestone  decisions1  and  reviews  within  the  Navy.  The 
conclusions  of  the  milestone  decisions  and  reviews  are  promulgated  in  an 
acquisition  decision  memorandum.  The  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition)  is  the  decision  authority  for  all 
acquisition  category2  (ACAT)  IC,  H,  and  m  programs.  The  Assistant  Secretary 
chairs  all  ACATIC,  II,  and  HI  NPDMs,  unless  otherwise  delegated.  The 
program  executive  officers,  direct  reporting  program  managers,  and  system 
commanders  are  delegated  the  decision  authority  for  all  ACAT  IV  programs. 

Role  of  Milestone  Decision  Authority.  At  each  decision  point,  the  milestone 
decision  authority  assesses  the  status  of  the  program  relative  to  the  user's  needs, 
the  established  program  baseline  and  acquisition  strategy,  and  approved 
financial  plans.  The  milestone  decision  authority  also  evaluates  the  updated 
acquisition  strategy  and  the  plans  for  conducting  the  next  phase  and  managing 
risk;  makes  cost-performance-schedule  trade-offs;  assesses  the  affordability  of 
what  is  being  proposed;  and  determines  whether  the  program  should  be 
terminated,  redirected,  or  allowed  to  continue  into  the  next  phase.  All 
operational  requirements,  testing,  funding,  and  the  majority  of  other  issues 
should  be  resolved  or  adequately  addressed  before  the  NPDM. 

Operational  Test  and  Evaluation.  The  DoD  Instruction  5000.2,  "Defense 
Acquisition  Policies  and  Procedures,"  February  23,  1991,  requires  that 
acquisition  programs  successfully  complete  a  dedicated  phase  of  operational  test 
and  evaluation  before  Milestone  ni,  Production  Approval.  The  dedicated  phase 
of  operational  test  and  evaluation  must  demonstrate  the  operational  effectiveness 
and  suitability  of  production  or  production-representative  units  under  realistic 
combat  conditions  to  determine  whether  operational  performance  requirements 
are  satisfied.  The  Director,  Operational  Test  and  Evaluation,  Office  of  the 
Secretary  of  Defense  (OSD),  must  approve  all  operational  test  plans  for 
ACAT  ID  and  IC  Programs.  The  Director  must  also  issue  a  beyond  low-rate 
initial  production  report  before  entry  into  full-rate  production  on  ACAT  I  and 


1The  point  when  a  recommendation  is  made  and  approval  sought  regarding  starting  or 
continuing  (proceeding  to  next  phase)  an  acquisition  program.  Milestones  are:  0  (Concept 
Studies  Approval),  I  (Concept  Demonstration),  II  (Development  Approval),  m  (Production 
Approval),  and  IV  (Major  Modification  Approval). 

^An  flfjptlglHon  category  is  a  classification  established  to  facilitate  decentralized  decisionmaking 
and  execution  and  compliance  with  statutorily  imposed  requirements.  The  categories  determine 
the  level  of  review,  decision  authority,  and  applicable  procedures. 
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other  designated  programs  on  the  adequacy  of  testing  conducted  by  the 
respective  independent  operational  test  and  evaluation  organization  and  whether 
the  test  results  confirm  operational  effectiveness  and  suitability. 

Consolidated  Automated  Support  System.  The  Navy's  CASS  Program  is  an 
ACAT II  Program  composed  of  standardized  automatic  test  equipment  (ATE) 
with  computer-assisted,  multi-function  capabilities  to  support  testing  of  aircraft 
subsystems  and  missiles.  The  CASS  Program  features  fleet-wide 
standardization  of  hardware  and  software  elements  and  is  designed  to  enhance 
electronic  test  capability  over  existing  ATE.  The  CASS  Program's  system 
hardware  is  also  designed  to  provide  the  Navy  intermediate  and  depot-level 
maintenance  with  better  electronic  testing  capability.  The  CASS  Program  is 
composed  of  functional  interfaces  allowing  the  testing  of  a  variety  of  hardware. 
The  interfaces  are  evolving  to  meet  Air  Force  as  well  as  Navy  requirements. 
To  minimize  unique  types  of  automatic  test  systems,  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  has  established  policy  to  require  DoD 
Components  to  satisfy  all  acquisition  needs  for  ATE  hardware  and  software  by 
using  designated  automatic  test  systems.  The  CASS  Program  is  one  of  those 
designated  systems.  The  Naval  Air  Systems  Command  is  assigned  lead  systems 
command  responsibility  for  the  CASS  Program  with  participation  from  the 
Naval  Sea  Systems  Command  and  the  Space  and  Naval  Warfare  Systems 
Command.  The  Navy  Acquisition  Executive  approved  low-rate  initial 
production3  (LRIP)  for  55,  60,  and  68  CASS  stations,4  in  September  1990,  July 
1992,  and  June  1993,  respectively.  NPDMs  for  Milestone  HI,  Production 
Approval,  decisions  for  a  Lot  IV  buy  of  63  CASS  stations  was  made  in  March 
1994;  Lot  V  and  Lot  VI  buys  of  60  and  82  CASS  stations  are  scheduled  for 
March  1995  and  FY  1996,  respectively.  The  Navy  plans  to  procure  a  total  of 
720  CASS  stations.  As  of  November  1993,  the  total  cost  of  the  CASS  Program 
was  estimated  at  about  $1.8  billion  in  FY  1990  constant  dollars  of  which 
$1.5  billion  and  $298.1  million  was  for  procurement,  and  research  and 
development,  respectively. 


Objective 

The  overall  audit  objective  was  to  evaluate  the  effectiveness  of  the  milestone 
review  process  for  Component-managed  acquisition  programs.  The  audit  also 
assessed  the  adequacy  of  the  information  provided  to  Military  Department 
milestone  decision  authorities  in  support  of  major  milestone  and  program 
reviews  and  evaluated  internal  controls  related  to  the  objective.  The  CASS 
Program  was  one  program  reviewed  during  the  audit.  During  the  audit  survey, 


^Low-rate  initial  production  is  the  production  of  a  system  in  limited  quantity  to  provide  articles 
for  operational  test  and  evaluation,  to  establish  an  initial  production  base,  and  to  permit  an 
orderly  increase  in  the  production  rate  sufficient  to  lead  to  full-rate  production  upon  successful 
completion  of  operational  testing. 

4A  CASS  station  is  a  collection  of  multi-function  ATE  structured  around  a  common  core  with 
four  testing  configurations  (Appendix  A). 
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we  determined  that  the  CASS  Program  was  scheduled  to  have  a  Milestone  HI, 
Production  Approval,  decision  before  the  completion  of  our  overall  audit  work. 
Therefore,  we  are  reporting  this  issue  separately  because  action  is  needed  on  the 
identified  issues  before  the  conclusion  of  our  overall  audit. 


Scope  and  Methodology 


We  conducted  this  program  audit  from  May  1993  through  February  1994  and 
reviewed  data  dated  from  February  19/6  through  February  1994.  To 
accomplish  the  objective,  we: 

o  discussed  issues  relating  to  the  effectiveness  of  the  milestone  review 
process  for  the  CASS  Program  with  OSD  and  Navy  personnel; 

o  determined  the  adequacy  of  the  information  that  the  Navy  provided  to 
the  decision  authorities  in  support  of  major  milestone  and  program  reviews; 

o  evaluated  the  effectiveness  of  the  milestone  review  and  program 
review  processes  for  the  CASS  Program;  and 

o  reviewed  CASS  Program  decision  documents  as  well  as  selected 
acquisition  reports,  Defense  acquisition  executive  summary  reports,  and  various 
contract  cost  management  reports. 

The  audit  was  made  in  accordance  with  auditing  standards  issued  by  the 
Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD,  and  accordingly  included  such  tests  of  internal  controls  as  were 
deemed  necessary.  We  did  not  rely  on  computer-processed  data  to  support  our 
finding  and  recommendations.  Appendix  D  lists  the  organizations  visited  or 
contacted. 


Internal  Controls 


Internal  Controls  Evaluated.  We  evaluated  internal  controls  related  to  the 
effectiveness  of  the  milestone  review  process  and  the  adequacy  of  the 
information  provided  to  the  milestone  decision  authorities  in  support  of  major 
milestone  and  program  reviews  for  the  CASS  Program.  The  DoD  Instruction 
5000.2,  "Defense  Acquisition  Policies  and  Procedures,"  February  23,  1991, 
and  DoD  Manual  5000.2-M,  "Defense  Acquisition  Management  Documentation 
and  Reports,"  February  23,  1991,  specify  those  controls  and  procedures.  We 
also  assessed  implementation  of  the  requirements  of  DoD  Directive  5010.38, 
"Internal  Management  Control  Program,"  April  14,  1987,  including 

performance  of  vulnerability  assessments  and  management  control  reviews. 
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Internal  Control  Weakness  Identified.  We  identified  a  material  internal 
control  weakness  as  defined  by  Office  of  Management  and  Budget 
Circular  A-123,  "Internal  Control  Systems,"  August  4,  1986,  and  DoD 
Directive  5010.38.  Internal  controls  to  conduct  milestone  reviews  existed; 
however,  those  controls  were  not  fully  effective.  Specifically,  the  Navy  had 
issued  policy  in  a  March  30,  1992,  memorandum  stating.  that  cost  and 
operational  effectiveness  analyses  (COEAs)  were  not  required  to  support 
Milestone  ffl,  Production  Approval,  decisions.  This  policy  was  contrary  to 
DoD  Instruction  5000.2  and  contributed  to  the  lack  of  a  formal  COEA  on  the 
CASS  Program.  The  weakness  was  not  reported  in  the  Naval  Air  Systems 
Command  Internal  Management  Control  Program  because  no  management 
control  reviews  were  performed  on  the  CASS  Program.  Our  summary  report 
will  address  the  overall  implementation  of  the  DoD  Internal  Management 
Control  Program  for  the  Milestone  Review  Process. 

Internal  Control  Weakness  Correction.  Recommendation  l.f.  in  this  report, 
if  implemented,  will  correct  the  internal  control  weakness.  Monetary  benefits 
associated  with  the  implementation  of  our  recommendation  to  require  COEAs  in 
support  of  Navy  Milestone  HI  decisions  are  not  quantifiable  because  benefits 
will  depend  on  future  actions  by  the  Navy.  A  copy  of  the  final  report  will  be 
provided  to  the  senior  official  responsible  for  internal  controls  in  the 
Department  of  the  Navy. 


Prior  Audits  and  Other  Reviews 


Since  1988,  the  General  Accounting  Office  and  the  Office  of  the  Inspector 
General,  DoD,  have  issued  one  and  five  reports,  respectively,  that  included  the 
CASS  Program.  However,  we  did  not  follow  up  on  the  prior  audit  reports 
because  they  did  not  contain  findings  or  recommendations  related  to  our 
objective. 


Other  Matters  of  Interest 


On  December  17,  1993,  the  Deputy  Inspector  General,  DoD,  sent  a 
memorandum  to  the  Navy  stating  concerns  about  the  readiness  of  the  CASS 
Program  for  production  and  deployment.  Although  our  audit  was  not  complete, 
we  had  cause  to  be  concerned  mat  a  full-rate  production  decision  might  be 
approved  prematurely.  We  worked  with  Navy  management  to  address  those 
concerns  and,  as  a  result,  the  Navy  deferred  an  NPDM  until  March  1994  and 
clarified  program  documentation  on  the  CASS  Program's  acquisition  strategy 
and  test  program.  Hie  Navy  actions  are  reflected  in  this  report. 

On  March  25,  1994,  the  Navy  held  an  NPDM  concerning  a  Milestone  HI, 
Production  Approval,  decision  for  the  CASS  Program.  The  NPDM  addressed 
the  recommendations  in  our  draft  report  on  the  CASS  Program.  The  Navy 
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concurred  with  Recommendations  l.a.  through  l.f.  that  were  addressed  to  it  and 
indicated  that  Recommendations  2.a.  and  2.b.,  addressed  to  the  Director, 
Operational  Test  and  Evaluation,  were  acceptable.  However,  neither  the 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  nor 
the  Director,  Operational  Test  and  Evaluation,  provided  formal  comments  to 
our  draft  report. 
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Part  II  -  Finding  and  Recommendations 


1 


Readiness  for  Acquisition  Milestone  HI, 
Production  Approval 

The  overall  CASS  Program  was  not  ready  to  proceed  into  the  Production 
and  Deployment  phase  of  the  acquisition  process.  'Die  CASS  Program 
management  had  not  successfully  completed  all  required  operational  test 
and  evaluations  and  verified  correction  of  deficiencies  identified  m 
testing  and  configuration  audits.  These  conditions  existed  because: 

o  the  CASS  Program  was  considered  to  be  low-risk, 

o  completion  of  testing  was  deferred  until  the  life-cycle  test 

phase, 


o  design  review  minutes  were  not  contract  deliverables, 

o  reporting  design  review  results  and  documenting  configuration 
changes  were  not  considered  a  priority,  and 

o  a  life-cycle  cost  estimate  was  considered  sufficient  cost  and 
operational  justification. 

As  a  result,  the  Navy  cannot  be  assured  that  the  CASS  Program  has  all 
deficiencies  corrected,  will  not  experience  excessive  rework  and  retrofit, 
and  will  not  require  additional  funding. 


Background 

Test  and  Evaluation  Guidance.  The  DoD  Instruction  5000.2,  part  8,  requires 
that  test  and  evaluation  verify  that  systems  have  attained  technical  performance 
specifications  and  are  operationally  effective  and  suitable  for  the  intended  use. 
The  DoD  requires  both  developmental  and  operational  testing  to  provide 
essential  information  to  support  decisionmaking.  Developmental  testing  verifies 
that  the  system  meets  technical  performance  specifications  and  that  it  is  ready 
for  operational  testing.  Operational  testing,  conducted  under  realistic 
conditions,  verifies  that  the  system  is  operationally  effective  and  suitable  for  the 
mission  intended  and  is  the  primary  method  of  predicting  system  performance. 

Configuration  Management  Guidance.  The  DoD  Instruction  5000.2,  part  9, 
section  A,  states  that  an  effective  configuration  management  program  will  be 
established  to  implement  the  decisions  made  in  the  systems  engineering  process 
by: 

o  documenting  and  verifying  the  functional  and  physical  characteristics 
of  a  configuration  item, 

o  controlling  changes  to  an  item  and  its  documentation, 
o  recording  the  configuration  of  actual  items,  and 
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o  auditing  the  configuration  item  and  its  configuration  identification. 

The  DoD  Instruction  5000.2,  part  6,  section  O,  attachment  1,  indicates  that 
design  change  activity  should  be  stabilized  by  demonstrating,  in  part,  that  a 
system  configuration  audit  has  been  accomplished  and  discrepancies  resolved. 

Cost  and  Operational  Effectiveness  Analysis  Guidance.  The  DoD 
Instruction  5000.2,  part  4,  and  DoD  Manual  5000.2-M,  part  8,  provide 
procedures  and  guidelines  for  COEAs.  The  COEA  evaluates  the  costs  and 
benefits,  including  operational  effectiveness  or  military  utility,  of  alternative 
actions  to  meet  recognized  Defense  needs.  The  COEA  aids  decisionmaking, 
facilitates  communications,  and  documents  acquisition  decisions  by  highlighting 
the  advantages  and  disadvantages  of  the  alternatives  being  considered  and 
showing  the  sensitivity  of  each  alternative  to  possible  changes  in  key 
assumptions,  such  as  variables,  including  selected  performance  capabilities. 
Further,  the  COEA  provides  early  identification  and  discussion  of  reasonable 
alternatives  among  decisionmakers  and  staffs  at  all  levels.  A  COEA  is  required 
to  be  prepared  and  considered  at  milestone  decision  reviews  of  ACATI 
programs  beginning  with  Milestone  I,  Concept  Demonstration  Approval.  At 
Milestone  n,  Development  Approval,  the  COEA  establishes  performance  floor 
and  cost  ceiling  objectives  or  acceptable  bands  for  possible  combinations  of  cost 
and  performance;  shows  the  trade-offs  used  to  arrive  at  the  objectives  for 
Phase  n,  Engineering  and  Manufacturing  Development;  and  examines  the 
impact  of  program  termination.  At  Milestone  in,  Production  Approval,  the 
COEA  may  update  the  previous  COEA  required  at  Milestone  II;  however,  if  the 
program  has  experienced  major  performance  or  cost  changes  during  Phase  n,  a 
new  COEA  may  be  required.  The  elements  of  the  updated  COEA  for  a 
Milestone  HI  review  will  be  specified  by  the  milestone  decision  authority  as  part 
of  the  pre-milestone  planning  process.  At  Milestone  IV,  Major  Modification 
Approval,  the  milestone  decision  authority  may  require  a  COEA  and  specify 
what  elements  to  include  in  the  COEA.  The  DoD  Component  responsible  for 
the  mission  area  in  which  a  deficiency  or  opportunity  is  identified  normally 
prepares  the  COEA.  The  DoD  Component  head  or  a  designee  determines  the 
independent  analysis  organization  to  prepare  the  COEA. 


Preparation  for  Milestone  III 

Prior  to  the  Milestone  m,  Production  Approval,  decision  in  March  1994,  the 
CASS  stations  had  not  demonstrated  through  developmental  and  operational 
testing  that  they  met  all  minimum  acceptable  performance  requirements  and 
contract  specifications.  Deficiencies  identified  by  the  testing  that  could  not  be 
corrected  were  deferred  to  future  life-cycle  testing.  The  Navy  is  changing  the 
CASS  stations  to  correct  the  deficiencies  and  is  producing  CASS  stations  and 
subsystems  containing  most  of  the  improvements.  However,  until  those  CASS 
stations  are  operationally  tested,  decisionmakers  will  not  know  whether  the 


5xhe  underlying  principles  and  analytical  concepts  of  a  COEA  are  tailored  and  implemented  in 
support  of  ACAT II,  III,  and  IV  programs  as  considered  appropriate  by  the  respective  DoD 
Component  acquisition  executive. 
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changes  have  resolved  the  identified  problems.  The  Navy  plans  to  conduct  a 
follow-on  operational  test  and  evaluation  on  CASS  stations  to  verify  that  the 
previously  identified  problems  have  been  resolved;  however,  this  follow-on 
operational  test  is  scheduled  to  be  conducted  after  a  full-rate  production 
decision.  The  Navy  has  contracted  for  183  CASS  stations.  Before  the  planned 
follow-on  operational  tests,  the  Navy  will  have  contracted  for  approximately 
306  CASS  stations,  which  is  about  43  percent  of  the  total  production  of 
720  stations. 

Technical  Evaluation.  From  February  5,  1992,  through  May  5,  1992,  the 
Commander,  Operational  Test  Evaluation  Force,  conducted  an  operational 
evaluation,  Operational  Testing  (OTVIIB,  of  the  four  CASS  configurations, 
including  hybrid,  radio  frequency,7  communication/navigation/identification 
(CNI),8  and  electro-optical.9  The  evaluation  was  to  determine  the  operational 
effectiveness  and  operational  suitability  of  the  CASS  Program  and  its  readiness 
for  fleet  introduction;  however,  the  test  did  not  expose  CASS  stations  to  the 
rigors  of  shipboard  operations,  one  of  the  intended  operating  environments. 
Five  test  stations,  including  two  hybrids,  one  with  a  pneumatic  function 
generator  (PFG);10  radio  frequency;  CNI;  and  electro-optical;  logged 
7,938  operating  hours  in  the  3-month  test  period.  A  total  of  156  faults  were 
inserted  to  test  the  CASS  station's  capability  to  detect  and  isolate  system  faults 
within  itself  and  in  test  program  sets  that  the  CASS  stations  interface  with. 

On  May  5,  1992,  Commander,  Operational  Test  Evaluation  Force,  concluded 
OT-TTB  and  recommended  in  Chief  of  Naval  Operations  Report  No.  3960-12, 
"Operational  Evaluation  of  CASS  Automatic  Test  Equipment,"  July  27,  1992, 
that  fleet  introduction  of  the  CASS  Program  be  delayed  until  12  specific 


6 The  hybrid  station  is  referred  to  as  the  "core"  of  the  total  CASS  test  system  and  as  the  "basic 
CASS  tester."  When  the  hybrid  station  is  combined  with  additional  instrumentation  and  control 
software,  three  additional  station  configurations  result.  Those  stations  are  the  radio  frequency, 
CNI,  and  the  electro-optical  (Appendix  A). 

7The  radio  frequency  station  has  semiautomatic  capabilities  to  test  avionic  units  whose 
technology  encompasses,  but  is  not  limited  to,  radar,  low  and  high-frequency  stimulus, 
measurement,  and  digital  functions. 

®The  CNI  station  has  semiautomatic  capabilities  to  test  avionic  units  whose  technology 
encompasses,  but  is  not  limited  to,  communications,  navigation,  identification,  low-frequency 
stimulus  or  measurement,  and  digital  functions. 

9 The  electro-optical  station  provides  for  electro-optical  performance  tests,  fault  isolation,  and 
functional  testing. 

l^The  PFG,  which  is  an  important  piece  of  ancillary  equipment,  is  a  pressure/vacuum 
measurement  and  control  system  used  to  measure,  control,  and  display  pressure  and  rates  of 
change  of  altitude  and  airspeed  associated  with  CASS  stations.  During  OT-HB,  the  PFG  proved 
to  be  immature  and  unreliable  because  so  many  PFG-specific  problems  were  identified  by  the 
test  team  that  the  PFG  had  to  be  excluded  in  order  to  proceed  with  further  testing.  These  PFG- 
specific  problems  do  not  allow  for  the  repair  of  the  PFG,  therefore,  reducing  the  maintainability 
of  the  CASS  stations.  The  Navy  is  redesigning  the  PFG  and  is  confident  about  the  PFG's  future 
performance;  however,  the  PFG  has  not  yet  been  tested  by  the  user  in  its  expected  operational 
environment. 
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problem  areas  are  corrected  and  the  corrections  are  verified  in  another  phase  of 
operational  test  and  evaluation.  Accordingly,  the  Naval  Air  System  Command 
tasked  the  Flight  Test  and  Engineering  Group,  Naval  Air  Warfare  Center 
Aircraft  Division,  Patuxent  River,  Maryland  (the  Flight  Test  and  Engineering 
Group),  to  perform  a  technical  evaluation.  Developmental  Testing  (DT)-IIC-3, 
on  production  versions  of  the  CASS  stations.  The  Naval  Air  Systems 
Command  limited  the  scope  of  DT-IIC-3  to  the  12  specific  problem  areas  and 
deferred  the  following  testing  to  a  CASS  Life-Cycle  Test  Program:11 

o  regression  testing  for  measuring  contract  compliance  with  electronic 
signals  at  the  input/output, 

o  regression  testing  for  measuring  contract  compliance  for  station 
control  software,  and 

o  deficiencies  identified  during  contractor  demonstration  testing. 

Also  deferred  to  the  Life-Cycle  Test  Program  were  32  deficiencies  identified 
during  DT-HC-2  testing  in  February  1992.  As  of  September  23,  1993,  14  of 
the  32  deficiencies  have  been  fully  or  partially  closed.  The  following  OT-IIB 
deficiencies  were  also  deferred  to  the  Life-Cycle  Test  Program: 

o  The  PFG  was  in  redesign  and  not  available  during  DT-IIC-3  testing. 

o  Streamlining  the  maintenance  documentation  system  would  avoid 
duplication  of  effort. 

On  September  30,  1992,  DT-DC-3  began  with  more  than  12,000  hours  of 
testing  performed  on  three  configurations  of  CASS  stations,  including  a  hybnd, 
radio  frequency,  and  electro-optical  stations  with  a  full-scale  engineering 
development  electro-optical  console.  The  CNI  and  the  hybrid  with  the  PFG 
configurations  that  were  tested  in  OT-IIB  were  not  tested  in  DT-IIC-3  because 
the  spread  spectrum  assets  that  make  up  the  CNI  were  not  available  and  the 
PFG  proved  to  be  immature  and  unreliable,  requiring  redesign.  The  technical 

llxhe  CASS  Life-Cycle  Test  Program  is  a  recent  innovation  for  supporting  the  Naval  Air 
Systems  Command  policy  on  program  management  responsibility  for  aviation  systems.  The 
CASS  Program  Manager  is  responsible  for  maintaining  readiness  and  performance  of  the  system 
and  continuous  improvement  throughout  the  product  life  cycle.  Those  responsibilities  are 
supported  through  an  on-going  test  and  measurement  process  to  monitor  station  performance  in 
the  field,  antioipnte  problems,  determine  technical  solutions,  and  drive  continuous  product 
improvement.  The  CASS  Life-Cycle  Test  Program  began  in  FY  1992  and  continues  throughout 
the  life  cycle  of  the  program. 

l^The  Maintenance  Documentation  System  was  ineffective  and  time-consuming.  The  procedure 
was  designed  to  collect  maintenance  data  for  possible  future  interface  with  the  Naval  Aviation 
Logistics  Command  Management  Information  System,  Phase  H;  the  intermediate  maintenance 
activity;  and  the  Supply  Support  Center  automated  maintenance  and  material  control  system. 
sinco  flie  user  would  have  to  enter  information  in  both  the  CASS  stations  and  again  in  the  Naval 
Aviation  Logistics  Command  Management  Information  System,  Phase  II,  the  maintenance 
documentation  system  tested  was  a  duplication  of  effort.  However,  duplication  of  maintenance 
documentation  in  the  CASS  Program  is  now  selectable  rather  than  mandatory,  eliminating 
redundancy. 
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and  operational  characteristics  and  thresholds  for  each  test  event  applied  to  the 
DT-HC-3  CASS  test  stations  were  not  as  stringent  as  those  applied  to  the 
OT-IIB  CASS  test  stations.  Comparison  of  the  test  and  evaluation  master  plans 
for  OT-IIB  and  DT-EC-3  showed  that  DT-EC-3  had  a  significant  reduction  in 
test  parameters  (Appendix  B)  and  eliminated  the  requirement  to  successfully 
complete  OT-EIA  before  full  fleet  introduction  of  the  CASS  Program.  The 
present  phase  of  OT-IE,  which  replaced  OT-EIA  in  the  test  plan,  is  less 
stringent.  Specifically,  the  OT-IEA  test  plan  required  that  "Successful 
accomplishment  of  OT-EIA  will  support  a  recommendation  for  full  fleet 
introduction."  This  statement  was  not  in  the  OT-E  test  plan.  The  purpose  of 
OT-IE  is  to  determine  the  operational  effectiveness  and  operational  suitability  of 
the  CASS  stations  to  accurately  detect  and  isolate  faults  within  units  under  test 
when  executing  maintenance  and  production  weapons  systems  test  program  sets, 
to  verify  correction  of  deficiencies  identified  in  OT-EC,  and  to  complete 
deferred  or  incomplete  operational  test  and  evaluation.  Therefore,  based  on  the 
current  acquisition  strategy,  operational  test  and  evaluation  of  the  test  program 
sets  to  detect  and  isolate  faults  in  weapon  replaceable  assemblies  will  not  be 
accomplished  until  after  Milestone  IE. 

During  the  DT-EC-3,  30  deficiency  reports  that  coincide  with  the  12  problem 
areas  from  the  OT-EB  were  written.  Of  the  30  deficiency  reports  written, 
21  were  closed,  8  were  moved  to  the  Life-Cycle  Test  Program,  and  1  was 
combined  with  another  deficiency  report  that  was  moved  to  the  Life-Cycle  Test 
Program.  Additionally,  198  quality-problem  reports  were  issued  to  the 
contractor  of  which  167  were  corrected.  The  remaining  31  quality-problem 
areas  were  considered  technically  low  risk  and  moved  to  the  Life-Cycle  Test 
Program.  In  the  Life-Cycle  Test  Program,  25  of  the  31  quality  problem  areas 
were  closed.  Of  the  remaining  6  problems  areas,  2  are  undergoing  verification 
by  the  Flight  Test  and  Engineering  Group  to  validate  correction  of  the 
problems,  1  was  submitted  to  the  CASS  Technical  Working  Group  for  approval 
to  proceed  with  deficiency  correction,  and  3  are  test  program  set  issues  that 
need  future  life-cycle  testing. 

On  May  14,  1993,  the  Flight  Test  and  Engineering  Group  concluded  that  the 
3  configurations  of  CASS  stations  used  in  the  DT-EC-3  and  all  associated 
software  were  technically  low  risk  and  could  proceed  to  OT-EC.  However,  the 
Flight  Test  and  Engineering  Group  recommended  further  testing  during  the 
Life-Cycle  Test  Program  to  validate  the  following  items  that  were  deferred  to 
the  Life-Cycle  Test  Program: 

o  two  OT-EB  deficiencies  noted  above, 

o  deficiencies  identified  during  DT-nC-2  testing, 

o  regression  testing  for  measuring  contract  compliance  with  electronic 
signals  at  the  input/output, 


l^Low  risk  means  that  the  program  has  a  high  probability  of  meeting  all  performance 
objectives;  ordinary  risk  associated  with  doing  business;  and  no  special  management  action  or 
planning  required  to  verify  compliance  with  cost,  schedule,  or  performance  requirements. 
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o  regression  testing  for  measuring  contract  compliance  for  station 
control  software,  and 

o  deficiencies  identified  during  contractor  demonstration  testing. 

Operational  Evaluation.  The  Navy  conducted  the  OT-HC  at  the  CASS  Test 
Facility,  Naval  Air  Station,  Patuxent  River,  Maryland,  and  aboard  the  USS  Carl 
Vinson.  The  purpose  of  the  test  was  to  determine  the  operational  effectiveness 
and  operational  suitability  of  the  CASS  Program  and  its  readiness  for  fleet 
introduction.  The  12  major  deficiencies  from  OT-HB  were  to  be  corrected  and 
the  corrections  verified  before  fleet  introduction  of  the  CASS  Program.  Those 
deficiencies  were  to  be  examined  during  the  OT-IIC  testing.  However,  the 
deficiencies  associated  with  the  PFG  could  not  be  examined  because  the  PFG 
was  undergoing  re-engineering  to  correct  the  deficiencies  experienced  in  OT-IIB 
and  was  not  available  for  OT-IIC  testing. 

From  May  31  through  July  9,  1993,  OT-IIC  logged  2,287  operating  hours  on 
3  configurations  of  CASS  stations,  including  a  hybrid,  radio  frequency,  and 
electro-optical,  at  the  CASS  Test  Facility.  The  technical  and  operational 
characteristics  and  thresholds  for  tests  applied  to  the  OT-IIC  CASS  test  stations 
were  not  as  stringent  as  those  applied  to  the  OT-HB  CASS  test  stations 
(Appendix  B).  From  June  22  through  30,  1993,  a  CNI-configured  CASS 
station  was  tested  aboard  the  USS  Carl  Vinson.  While  the  stations  at  the  CASS 
Test  Facility  were  configured  with  production-representative  software  and 
associated  firmware,  the  station  aboard  the  ship  was  configured  with  non¬ 
production-representative  software  to  run  a  test  program  set  in  support  of  the 
Joint  Tactical  Information  Distribution  System  developmental  testing.  As  a 
result,  reliability  data  was  not  collected  on  the  shipboard  CNI  station  because  of 
the  difference  in  software  and  firmware  from  that  at  the  shore  stations. 

On  September  29,  1993,  the  Commander,  Operational  Test  and  Evaluation 
Force,  reported  the  results  of  the  OT-HC.  The  report  did  not  specifically 
address  whether  the  12  major  OT-HB  deficiencies  were  satisfactorily  corrected; 
however,  it  did  state  that  production  CASS  stations  are  operationally  effective, 
operationally  suitable  in  the  shore  environment,  and  potentially  operationally 
suitable  in  the  afloat  environment.  The  report  recommended  fleet  introduction 
to  ashore  facilities  and  supported  shipboard  installations  when  the  following 
deficiencies  were  corrected  to  the  Chief  of  Naval  Operations'  satisfaction: 

o  full  evaluation  of  the  impact  of  shipboard  Electromagnetic 
Interference  and  vibrations  on  CASS  station  operations  and 

o  redesign  the  unit  under  test  holding  fixture  to  conform  with  shipboard 
space  limitations. 

The  report  also  noted  that,  when  shipboard  CASS  station  installations  become 
fully  operational,  shipboard  interoperability  of  the  CASS  Program  will  need  to 
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be  evaluated  in  later  test  phases.  The  need  for  further  evaluation  occurred 
because  CASS  station  interoperability14  was  tested  in  the  ashore  operating 
environment  only. 

On  November  4,  1993,  the  Director,  Air  Warfare,  Office  of  the  Chief  of  Naval 
Operations,  issued  a  memorandum  addressing  the  OT-HC  report.  The 
memorandum  stated  that,  based  on  assessments  by  engineers  from  the  Flight 
Test  Engineering  Group,  the  issues  concerning  the  impact  of  shipboard 
electromagnetic  interference,  vibrations  on  CASS  station  operations,  and  unit 
under  test  holding  fixture  were  resolved;  therefore,  the  CASS  Program  has 
demonstrated  its  readiness  for  full  fleet  introduction.  The  assessments  indicated 
that  the  engineers  tested  the  impact  of  shipboard  electromagnetic  interference 
and  vibrations  on  CASS  station  operations,  found  those  concerns  to  be  low  risk, 
and  recommended  closure.  However,  for  the  unit  under  test  holding  fixture 
deficiency,  the  engineers  did  not  do  any  operational  testing  to  determine 
whether  the  discrepancy  was  corrected.  The  engineers  based  their 
recommended  closure  of  the  discrepancy  on  corrections  being  incorporated  into 
the  system  and  assumed  that  the  corrections  would  be  effective  without  doing 
additional  testing  to  verify  that  the  deficiency  was  corrected. 

Scheduling  Milestone  Decision.  Based  on  the  results  of  OT-IIC  and  the 
approval  by  the  Office  of  the  Chief  of  Naval  Operations,  the  CASS  Program 
was  scheduled  for  a  Milestone  in,  Production  Approval,  decision  in  March 
1994.  After  we  issued  our  December  17,  1993,  memorandum,  the  Navy 
clarified  that  the  Milestone  HI  decision  was  only  on  3  configurations  of  CASS 
stations,  including  the  hybrid,  radio  frequency,  and  CNI.  Another  Milestone  III 
decision  for  the  electro-optical  station  configuration  is  not  scheduled  until 
March  1995  after  the  completion  of  additional  testing.  We  found  no  reference 
to  a  second  Milestone  m  decision  on  the  electro-optical  station  configuration 
until  the  CASS  Program  Manager  issued  a  memorandum  February  16,  1994, 
that  revised  the  Integrated  Program  Summary,  stating  the  recommendation  to 
the  Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
as: 


Approve  Milestone  HI  for  the  CASS  Hybrid,  RF  [Radio  Frequency] 
and  CNI  configurations.  The  first  production  unit  (LR1P  H)  Electro¬ 
optic  (EO)  configuration  is  undergoing  Physical  Configuration  Audit 
(PCA)  now.  It  is  scheduled  for  delivery  in  November  1994  after 
completion  of  First  Article  and  Production  Acceptance  testing.  A 
Milestone  IH  decision  for  the  EO  configuration  will  be  requested  in 
2QTR  [second  quarter]  FY  1995. 

Based  on  uncertainties  as  to  whether  the  12  major  OT-IIB  deficiencies  and  the 
holding  fixture  deficiency  were  satisfactorily  corrected  and  the  need  for  further 
evaluation  concerning  the  electro-optical  and  shipboard  interoperability  of  CASS 
stations,  decisionmakers  cannot  be  fully  assured  that  the  overall  CASS  Program 
is  operationally  ready  for  fleet  introduction  and  for  full-scale  production. 
However,  we  consider  the  Navy  decision  to  hold  a  second  Milestone  HI  review 


^Interoperability  is  the  ability  to  adequately  interface  with  other  specific  ancillary  equipment 
and  other  systems  in  its  intended  operating  environment. 
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on  the  electro-optical  configuration  to  be  reasonable  so  that  the  tested  hybrid, 
radio  frequency,  and  CNI  configurations  can  proceed  as  scheduled  into 
production. 

Field  Reliability  Report.  The  Naval  Air  Warfare  Center,  Aircraft  Division, 
Lakehurst,  New  Jersey  (the  Aircraft  Division),  conducts  reliability  analyses  of 
LRIP  CASS  stations  installed  in  the  field.  A  report  on  the  analyses  is  updated 
monthly  and  distributed  to  the  CASS  Program  Office.  The  report  is  critical  to 
the  CASS  Program's  development  and  production  and  the  Navy  ATE  program 
and  is  an  excellent  management  tool.  The  report: 

o  provides  a  profile  of  system  performance, 

o  identifies  reliability  problem  areas, 

o  provides  direction  for  reliability  improvement, 

o  assists  the  Navy  in  reassessing  its  acquisition  strategy  for  the  best 
products, 

o  provides  maintainability  and  maintenance  information  for  the  CASS 
Program's  logistics  planning,  and 

o  provides  Navy  ATE  program  information. 

The  January  26,  1994,  report  indicated  that 

Over  the  past  few  months,  the  Digital  Test  Unit  (DTU)  Channel 
Cards  have  been  the  most  troublesome  CASS  asset.  This  fact  is 
reflected  in  the  last  two  reports.  Based  on  this  anomaly,  the  CASS 
Technical  Working  Group  (TWG)  has  concluded  that  an  improvement 
in  the  manufacturing  process,  especially  in  the  Environmental  Stress 
Screening  (ESS),  could  significantly  improve  the  reliability  of  the 
DTU  Channel  Cards.  While  the  vendor  of  the  DTU  Channel  Cards 
has  in  the  past  overlooked  the  ESS  process  due  to  "cost"  and 
"schedule"  consideration,  it  is  being  realized  that  an  appropriate  ESS 
process  is  relatively  cost  effective.  A  redesign  of  the  DTU  Channel 
Cards  is  also  an  option  offered  by  the  vendor.  Furthermore,  we  (the 
Aircraft  Division]  recommend  that  the  Navy  look  seriously  into  the 
rescreening  process  and  determine  if  a  more  stringent  rescreening 
process  is  also  cost  effective  for  the  CASS's  life  cycle. 

The  report  stated  that  digital  test  unit  failures  continue  to  dominate  test  results; 
however,  proper  attention  was  given  to  the  digital  test  unit  problem  during  the 
CASS  Reliability  and  Maintainability  Quarterly  Review  in  December  1993  and 
during  the  CASS  Technical  Working  Group  Meeting  in  January  1994.  The 
Navy  should  continue  close  scrutiny  of  field  reliability  reports  to  identify  and 
correct  deficiencies. 
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Design  Review  and  Functional  and  Physical  Configuration 
Audit15  Deficiencies 


The  Navy  did  not  verify  that  design  review  deficiencies  were  documented  and 
that  discrepancies  identified  during  the  functional  and  physical  configuration 
audits  were  resolved  before  the  scheduled  Milestone  HI  decision. 

Design  Reviews.  The  CASS  Program  Office  conducted  the  Critical  Design 
Review15  and  the  Final  Design  Review17  in  1986  and  1987,  respectively. 
However,  design  deficiencies  were  not  documented  and  tracked  to  validate  that 
configuration  control  was  maintained  throughout  the  life  cycle  of  the  program. 
Further,  without  documented  deficiencies,  die  Navy  could  not  verify  during  the 
PCA  and  FCA  that  deficiencies  were  resolved  and  incorporated  in  the  design. 

Critical  Design  Review.  The  Critical  Design  Review  was  used  to 
conduct  source  selection  rather  than  to  perform  an  in-depth  technical  assessment 
of  design  maturity  or  to  document  deficiencies  and  establish  an  audit  trail  to 
substantiate  changes.  During  the  Critical  Design  Review,  the  CASS  Program 
Office  evaluated  the  proposals  of  General  Electric,  Aerospace  Division,  and 
Grumman  Aerospace  Corporation  to  select  one  contractor,  which  was  General 
Electric,  Aerospace  Division. 


^Military  Standard  973,  "Configuration  Management,"  April  17,  1992,  defines  a  functional 
configuration  audit  (FCA)  as  the  formal  examination  of  functional  characteristics  of  a 
configuration  item,  before  acceptance,  to  verify  that  the  item  has  achieved  the  requirements 
specified  in  its  functional  and  allocated  configuration  documentation.  A  physical  configuration 
audit  (PCA)  is  the  formal  examination  of  the  as-built  configuration  of  an  item  against  its  design 
documentation.  The  PCA  for  a  configuration  item  shall  not  be  started  unless  the  FCA  for  the 
configuration  item  has  already  been  accomplished  or  is  being  accomplished  concurrent  with  the 
PCA.  The  PCA  includes  a  detailed  audit  of  engineering  drawings;  specifications;  technical  data; 
tests  used  in  production  of  configuration  items;  and  design  documentation,  listings,  and 
operation  and  support  documents  for  computer  software  configuration  items.  Preliminary  and 
critical  design  review  minutes  shall  be  examined  to  verify  that  all  findings  have  been 
incorporated  and  completed. 

^Military  Standard  1521,  "Technical  Review  and  Audits  for  Systems,  Equipments,  and 
Computer  Software,"  June  1985,  states  that  a  critical  design  review  is  conducted  for  each 
configuration  item  when  detail  design  is  essentially  complete.  The  purpose  of  die  review  is  to 
determine  that  the  detail  design  of  the  configuration  item  under  review  satisfies  the  performance 
and  engineering  specialty  requirements  of  the  hardware  configuration  item  development 
specifications;  establish  detail  design  compatibility  among  the  configuration  item  and  other  items 
of  equipment,  facilities,  computer  software,  and  personnel;  assess  configuration  item  risk  areas 
and  the  results  of  the  producibility  analyses  conducted  on  system  hardware;  and  review  die 
preliminary  hardware  product  specifications.  For  computer  software  configuration  items,  the 
review  focuses  on  the  determination  of  the  acceptability  of  the  detailed  design,  performance,  and 
test  characteristics  of  the  design  solution  and  on  the  adequacy  of  the  operation  and  support 
documents. 

17The  Final  Design  Review  reviewed  changes  made  to  the  system  specifications  baseline  since 
the  Critical  Design  Review  in  1986. 
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Final  Design  Review.  The  Final  Design  Review  included  a  presentation 
of  the  design  by  General  Electric;  however,  deficiencies  and  appropriate  action 
items  were  not  prepared  and  tracked  through  closure.  CASS  Program  Office 
officials  stated  that  no  deficiencies  were  documented  since  the  contractor's 
design  met  or  exceeded  all  design  and  performance  requirements.  However, 
significant  redesign  work  was  performed  on  CASS  stations  to  change  them  to  a 
ducted  air  conditioning  cooling  system.  Also,  additional  redesign  work  remains 
on  the  under  unit  test  holding  fixture,  test  program  set  holding  fixture,  and  other 
CASS  assets.  The  CASS  Program  Office  questioned  the  contractor  about  air 
conditioning  cooling,  station  shut  down,  cable  routing,  and  other  deficiencies  at 
the  Final  Design  Review;  however,  the  Program  Office  did  not  prepare  action 
items  to  correct  the  deficiencies.  Those  deficiencies  caused  failures  that  were 
documented  as  deficiencies  during  developmental  testing.  Further,  the 
problems  were  also  identified  as  deficiencies  during  the  PCA.  Therefore,  the 
Navy  should  have  documented  and  tracked  deficiencies  at  the  Final  Design 
Review  to  correct  deficiencies  during  the  early  phase  of  development. 

Functional  Configuration  Audit  Results.  The  FCA  for  the  CASS  electronic 
warfare,  radar,  CNI,  electro-optical,  and  display  test  system  end  items  was 
conducted  incrementally  beginning  in  March  1990.  The  audit  documented 
259  discrepancies  of  which  12  remained  open  as  of  February  4,  1994. 
Examples  of  the  discrepancies  include  functions  related  to  the  electro-optical 
configuration  that  were  not  demonstrated  such  as  the  ability  to  measure 
divergence  of  the  laser,  minimum  and  maximum  energy  of  the  laser,  and 
amplitude  and  pulse  stability.  The  electro-optical  configuration  is  scheduled  for 
a  Milestone  m  decision  in  March  1995.  The  Navy  had  planned  to  procure 
6  LRIP  units  in  the  electro-optical  configuration  as  part  of  the  Lot  V  buy  to 
occur  in  FY  1995,  but  changed  plans  as  a  result  of  our  audit  to  delay  this 
procurement  until  die  Lot  VI  buy  in  FY  1996. 

Physical  Configuration  Audit  Results.  The  PCA  was  performed 
incrementally,  beginning  January  12,  1992.  Each  increment  of  the  PCA 
resulted  in  a  separate  fisting  of  deficiencies.  General  Electric,  Aerospace 
Division,  which  was  acquired  by  Martin  Marietta,  did  not  respond  to  the 
discrepancies  until  May  26,  1992,  after  the  fifth  PCA.  The  incremental  PCAs 
included  hardware  and  software  audits.  The  PCA  on  the  electro-optical 
configuration  was  not  completed,  but  is  scheduled  to  be  completed  leading  to 
completion  of  first  article  and  production  acceptance  testing  in  November  1994. 

Hardware  Audit.  The  Naval  Air  Technical  Services  Facility 
Detachment,  Washington,  District  of  Columbia  (the  Facility  Detachment),  that 
performed  the  hardware  PCA,  documented  675  hardware  deficiencies. 
Deficiencies  identified  during  the  hardware  PCA  included  incorrect  part 
markings,  components  installed  on  the  hardware  that  did  not  correspond  with 
the  drawing,  drawing  discrepancies,  drawings  that  did  not  correctly  represent 
the  hardware  configuration,  quantity  differences  on  the  parts  fist  as  compared  to 
the  hardware,  components  missing  from  the  hardware,  and  no  part  marking. 
On  November  15,  1993,  the  Facility  Detachment  stated  that  for  474  of  the 
675  deficiencies,  it  agreed  with  the  solutions  proposed  by  Martin  Marietta.  For 
the  remaining  201  deficiencies,  the  Facility  Detachment  disagreed  with  the 
solutions  for  32,  required  additional  information  for  161,  requested  solutions  for 
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2,  and  requested  resubmission  of  the  solution  for  6.  On  February  4,  1994,  die 
Facility  Detachment  stated  that  it  agreed  with  the  solutions  for  74  deficiencies 
and  requested  resubmission  of  the  solutions  for  6  deficiencies. .  However,  the 
Facility  Detachment  did  not  address  the  status  of  the  remaining  121  open 
deficiencies.1®  On  February  7,  1994,  in  a  point  paper,  the  CASS  Program 
Office  indicated  that  only  7  deficiencies  remained  open. 

Software  Audit.  The  Naval  Air  Warfare  Center,  Aircraft  Division, 
Lakehurst,  New  Jersey  (the  Aircraft  Division),  that  performed  the  software 
PCA,  documented  1,144  software  deficiencies.  As  of  November  30,  1993,  the 
Aircraft  Division  considered  1,058  of  the  1,144  deficiencies  to  be  closed, 
62  deficiencies  were  in  review,  and  24  deficiencies  relating  to  hardware 
specifications  were  awaiting  a  response  from  Martin  Marietta.  As  of 
January  28,  1994,  the  Aircraft  Division  stated  that  all  problems  generated 
during  the  software  PCA  have  been  successfully  resolved  and  closed.  Some 
problems  identified  during  the  PCA  included  no  documentation  for  the 
firmware,  no  delivery  of  software  programmer's  manuals  for  commercial  off- 
the-shelf  software,  and  no  delivery  of  other  software  documentation,  winch  is 
needed  to  validate  proper  support  of  the  equipment.  The  PCA  deficiencies 
resulted,  in  part,  from  a  lack  of  CASS  Program  software  licenses,  a  software 
license  issue  to  rebuild  all  computer  software  configuration  items  with  Martin 
Marietta  software,  and  a  compiler  license  for  the  Spread  Spectrum 
Modulator/Demodulator  firmware. 


Navy  Cost  and  Operational  Effectiveness  Analysis 


The  Navy  did  not  prepare  a  COEA  in  support  of  Milestones  HA-2  and  HA-3, 
Development  Approval  for  the  CASS  Program's  LRIP  Lots  II  and  HI  milestone 
decisions  in  July  1992  and  June  1993,  respectively,  and  the  Milestone  m, 
Production  Approval,  decision  scheduled  for  March  1994.  Instead,  for 
Milestones  HA-2  and  HA-3,  updated  program  life-cycle  cost  estimates  approved 
April  3,  1992,  and  May  27,  1993,  respectively,  and  a  cost  benefits  analysis 
were  substituted  for  COEAs.  The  life-cycle  cost  estimates  provided  the  total 
cost  to  the  Government  of  acquisition  and  ownership  of  the  CASS  Program  over 
its  useful  life,  including  the  cost  of  development,  acquisition,  and  support. 
However,  those  cost  estimates  lacked  comprehensive  cost  and  operational 
analyses  provided  by  a  COEA.  Prospective  Computer  Analysts,  Incorporated, 
originally  developed  the  cost  benefits  analysis  in  November  1986  to  support  the 
Milestone  I  decision  for  engineering  development  of  the  CASS  Program.  In 
August  1987  and  May  1992,  the  analysis  was  updated  to  support  the 
Milestone  E  and  m  decisions,  respectively.  The  analysis  extensively  considered 
four  alternatives  and  concluded  that  the  CASS  alternative  was  the  most  effective 
and  cost  efficient  approach  to  developing  and  operating  a  system  that  meets  the 
Navy's  ATE  mission  and  performance  goals  for  the  1990s  and  beyond. 


l^The  121  open  hardware  deficiencies  was  calculated  by  subtracting  the  74  agreed  to  solutions 
and  6  resubmission  solutions  from  the  remaining  201  deficiencies. 
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However,  the  cost  benefit  analysis  report  stated  that,  even  though  many 
elements  of  a  COEA  were  in  the  cost  benefit  analysis,  additional  efforts  were 
needed  to  provide  a  complete  COEA.  Those  additional  efforts  included: 

o  sensitivity  analysis  of  independent  cost  and  technical  variables; 

o  expansion  and  incorporation  of  mission  need  analyses; 

o  expansion  and  incorporation  of  threat  and  operational  environment 
statements; 

o  expansion  of  the  cost  uncertainty  analysis  of  the  cost  benefit  analysis; 

o  definition  of  cost  and  technical  thresholds; 

o  analysis  of  long-term  life-cycle  costs,  especially  operations  and 
support  costs  for  the  CASS  Program  and  the  Augmentation  of  Family  of  Testers 
alternative;  and 

o  reformatting  of  the  cost  benefit  analysis  results  and  additional  analyses 
above  into  the  format  specified  in  DoD  Manual  5000.2-M. 

In  view  of  the  comments  by  Prospective  Computer  Analysts,  Incorporated,  in 
the  May  1992  cost  benefit  analysis  stating  that  additional  efforts  needed  to  be 
performed  to  provide  a  complete  COEA,  we  believe  that  a  review  of  the 
analyses  should  be  conducted  to  verify  that  the  analyses  contain  the  elements  of 
a  COEA  needed  to  support  the  Milestone  HI,  Production  Approval,  decision. 


Cause  for  Inadequate  Accomplishment  of  Specific  Exit 
Criteria 


Preparation  for  the  Milestone  HI  decision  was  not  fully  adequate  because  the 
CASS  Program  was  considered  to  be  low-risk,  completion  of  testing  was 
deferred  until  the  life-cycle  test  phase,  design  review  minutes  were  not  contract 
deliverables,  reporting  design  review  results  and  documenting  configuration 
changes  were  not  considered  a  priority,  and  a  life-cycle  cost  estimate  was 
considered  sufficient  cost  and  operational  justification.  The  CASS  Program 
management  deferred  essential  Milestone  HI  prerequisites  until  after  the 
Milestone  in  decision  because  of  concerns  about  awarding  further  LRIP 
contracts  and  possible  breaks  in  production.  However,  the  program  risk 
associated  with  this  approach  should  be  clearly  presented  to  the  milestone 
decision  authority. 

Low-Risk  Program.  CASS  stations  were  not  tested  because  the  CASS 
Program  Office  considered  the  system  to  be  low  risk  and  deferred  testing  until 
the  life-cycle  test  phase  that  begins  after  Milestone  HI  Review  and  because 
CASS  stations,  subsystems,  and  ancillary  equipment  were  not  available  during 
operational  testing  and  evaluation. 
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Configuration  Audits  and  Design  Review.  Discrepancies  identified  during  the 
functional  and  physical  configuration  audits  were  not  fully  resolved  and  critical 
design  review  deficiencies  were  not  documented  because  CASS  Program 
officials  indicated  that  minutes  from  the  Critical  Design  Review  were  not  a 
deliverable  under  the  contract  and  because  the  risk  of  proceeding  to 
Milestone  in  with  correction  of  the  deficiencies  identified  but  untested  was 
considered  low  risk.  Further,  CASS  Program  officials  did  not  consider 
reporting  the  results  from  the  Final  Design  Review  to  management  and 
documenting  resolution  and  closure  to  be  a  high  priority. 

Life-Cycle  Cost  Estimate.  A  COEA  was  not  prepared  in  support  of  the 
Milestone  n  and  III  decisions  because  life-cycle  cost  estimates  were  considered 
sufficient  cost  and  operational  justification  to  support  acquisition  of  the  CASS 
Program  and  DoD  guidance  was  interpreted  as  not  requiring  a  COEA. 

Milestone  H.  Based  on  a  March  30,  1992,  memorandum  by  the  Office 
of  the  Assistant  Secretary  of  Navy  (Research,  Development  and  Acquisition) 
the  Navy  considered  a  life-cycle  cost  estimate  to  be  sufficient  instead  of 
preparing  a  COEA  to  support  the  milestone  review  process.  The  memorandum 
stated  that 


In  lieu  of  a  Cost  and  Operational  Effectiveness  Analysis  (COEA),  an 
updated  program  life  cycle  cost  estimate  will  suffice  for  the  scheduled 
intermediate  milestone  for  the  CASS  low  rate  initial  production 
(Lot  II,  Milestone  HA-2).  The  Navy  Program  Decision  Meeting  is 
scheduled  for  30  April  1992. 

The  memorandum  was  also  the  basis  for  not  preparing  a  COEA  for  the 
Milestone  HA-3  decision.  The  memorandum  was  based,  in  part,  on: 

o  an  updated  cost  benefit  analysis  prepared  for  Milestone  II A- 1  in 
September  1990  showed  that  the  CASS  Program  was  more  cost  beneficial  than 
upgrading  the  existing  family  of  testers  or  peculiar  test  equipment  alternatives, 

o  an  updated  life-cycle  cost  estimate  had  already  been  prepared  to 
support  the  Milestone  H-2A  decision,  and 

o  a  COEA  proposal  for  Milestone  III  had  been  prepared. 

The  COEA  proposal  prepared  by  the  CASS  Program  Office  stated,  in  part,  that 

The  CASS  Program  Manager  anticipates  requesting  approval  for  a 
second  Low  Rate  Initial  Production  (LRIP)  buy  in  April  92.  To 
support  this  Milestone  an  updated  program  Life  Cycle  Cost  (LCC) 

Estimate  is  viewed  as  sufficient  for  meeting  the  milestone  COEA 
requirement.  As  part  of  this  COEA  process,  the  LCC  will  be 
reviewed  and  its  results  and  assumptions  validated  by  the  COEA 
Study  Team  prior  to  Milestone  HA-2. 

Milestone  HI.  For  Milestone  III,  the  Office  of  the  Assistant  Secretary 
of  the  Navy  (Research,  Development  and  Acquisition)  determined  that  a  COEA 
was  not  required  based  on  a  previous  decision  and  interpretation  of  DoD 
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Instruction  5000.2  and  DoD  Manual  5000.2-M  even  though  an  undated  COEA 
proposal  by  the  CASS  Program  Office  indicated  that  a  COEA  was  required. 
The  COEA  proposal  stated,  in  part,  that 

Tn  preparation  for  Milestone  III,  a  complete  COEA  in  accordance  with 
the  requirements  of  reference  (a)  [DoD  Instruction  5000.2,  part  4, 
section  E,  "Cost  and  Operational  Effectiveness  Analysis"]  will  be 
accomplished  by  the  Study  Team.  The  team  will  study  and  analyze 
three  potential  solutions  to  the  Navy*s  Automatic  Test  Equipment 
requirement  for  the  latter  twentieth/early  twenty-first  century. 


Documentation  provided  by  the  CASS  Program  Office  indicated  that  the  COEA 
proposal  was  not  approved  and  that  a  COEA  was  not  required  for  the 
Milestone  IE  decision  based  on  the  March  30,  1992,  memorandum  by  the 
Office  of  the  Assistant  Secretary  of  Navy  (Research,  Development  and 
Acquisition),  which,  as  discussed  above,  addressing  only  an  LRIP  decision  and 
not  the  Milestone  HI  decision.  The  documentation  included  another 
memorandum  from  the  Office  of  the  Assistant  Secretary,  dated  October  9, 
1991,  and  one  from  the  Office  of  the  Deputy  Chief  of  Naval  Operations,  dated 
June  1,  1992,  that  also  indicated  that  a  COEA  was  not  required  for  the 
Milestone  in  decision.  Since  a  COEA  was  not  approved  for  the  Milestone  IE 
decision,  the  CASS  Program  Office  requested  studies  of  the  CASS  Program. 

Office  of  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition).  An  October  9,  1991,  memorandum  by  the 
Office  of  the  Assistant  Secretary  of  the  Navy  (Research,  Development  and 
Acquisition)  indicated  that  a  Milestone  m  does  not  normally  require  a  COEA. 
Instead,  the  program  office  would  provide  updated  life-cycle  cost  data,  as 
appropriate.  The  memorandum  indicated  that  the  decision  was  based  on  DoD 
Manual  5000.2-M,  part  8,  concerning  Milestone  in  guidance  and  stated  that 

A  cost  and  operational  effectiveness  analysis  is  not  required  unless 
conditions  have  changed  sufficiently  so  that  previous  cost- 
effectiveness  determinations  are  no  longer  valid.  Because  costs  are 
more  likely  to  have  changed,  Milestone  HI  analyses  often  provide 
updated  estimates  of  life  cycle  costs. 

We  disagree  with  the  above  position  by  the  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition).  If  costs  change,  the  validity  of  a 
previous  cost  and  operational  effectiveness  determination  can  only  be  assessed 
through  an  updated  COEA,  not  updated  life-cycle  costs.  _  Additionally,  since  a 
COEA  was  not  prepared  for  the  CASS  Program  at  Milestone  HI,  a  COEA 
cannot  be  updated.  Also,  DoD  policy  states  memoranda  such  as  this  by  the 
Assistant  Secretary  expire  if  not  formally  implemented  into  policy  within 
90  days.  A  COEA  should  have  been  prepared  for  the  CASS  Program's 
Milestone  IE  decision  to  determine  whether  the  Program  was  cost-effective. 
Since  a  COEA  had  not  been  prepared  for  the  previous  milestones, 
decisionmakers  could  not  validate  previous  COEA  determinations  and  assess 
whether  the  CASS  Program  was  the  most  cost-effective  alternative.  Further, 
DoD  Manual  5000.2-M  supplements  guidance  provided  by  DoD  Instruction 
5000.2.  The  Instruction  states  that 
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At  Milestone  in,  Production  Approval,  the  COEA  may  be  only  an 
update  of  the  Milestone  II  analysis.  However,  if  there  have  been 
major  performance  or  cost  changes  during  Phase  IE,  Engineering  and 
Manufacturing  Development,  a  new  analysis  may  be  required.  The 
ftlpmpntfi  of  die  analysis  to  be  updated  for  a  Milestone  III  review  will 
be  specified  by  the  milestone  decision  authority  as  part  of  the  pre- 
milestone  planning  process. 

Particularly  since  a  COEA  had  not  been  prepared  for  the  Milestone  II  decision, 
the  Assistant  Secretary  of  die  Navy  (Research,  Development  and  Acquisition) 
should  have  considered  requiring  the  preparation  of  a  COEA  as  part  of  the  pre- 
milestone  planning  process  for  the  CASS  Program's  Milestone  HI  decision  to 
validate  that  the  most  cost  and  operationally  effective  program  is  being 
produced  to  meet  recognized  Defense  needs. 

Office  of  the  Deputy  Chief  of  Naval  Operations.  A  June  1, 
1992,  memorandum  by  the  Office  of  the  Deputy  Chief  of  Naval  Operations  for 
Navy  Program  Planning19  interpreted  DoD  Instruction  5000.1,  "Defense 
Acquisition,"  February  23,  1991,  to  state  that,  beyond  Milestone  H,  a  COEA 
was  not  required  for  programs.  The  memorandum  stated  that  program 
managers  and  sponsors  should  be  notified  that  a  COEA  was  not  required.  The 
memorandum  indicated  that  representatives  from  the  Office  of  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  and  the  Office 
of  the  Deputy  Chief  of  Naval  Operations  made  the  determination.  Such  a 
determination  is  not  in  accordance  with  DoD  Instruction  5000.2,  as  noted 
above.  The  Office  of  the  Deputy  Chief  of  Naval  Operations  should  have 
recommended  that  an  updated  COEA  be  prepared  for  Milestone  in  decisions. 

Studies.  Since  the  preparation  of  a  COEA  was  not  approved  for 
the  Milestone  m  decision,  the  CASS  Program  Office  requested  studies  by  the 
Center  for  Naval  Analyses  and  the  Institute  for  Defense  Analyses. 

Center  for  Naval  Analyses  Study.  In  May  1993,  the 
Center  for  Naval  Analyses  issued  a  study  on  the  benefits  of  the  CASS  on  an 
aircraft  carrier.  The  study  examined  what  impact  the  replacement  of  existing 
ATE  with  CASS  stations  would  have  on  the  aircraft  intermediate  maintenance 
department  of  an  aircraft  carrier.  The  study  team  visited  the  USS  America; 
inventoried  the  ATE  that  was  aboard;  and  talked  with  aircraft  intermediate 
maintenance  department  officers,  shop  supervisors,  and  operators  about  tester 
usage  rates.  The  study  team  determined  the  number  and  type  of  CASS  stations 
that  would  be  required  to  produce  the  same  throughput  of  work  as  the  existing 
ATE.  The  study  concluded  that  by  replacing  existing  ATE  on  the  USS  America 
with  CASS  stations,  a  reduction  in  the  following  would  result: 

o  the  number  of  operators  and  maintainers  would 
drop  from  98  to  68,  saving  about  $1  million; 

o  associated  annual  training  costs  would  decrease 

by  a  factor  of  five;  and 

19Name  changed  to  Office  of  the  Deputy  Chief  of  Naval  Operations  for  Resources,  Warfare 
Requirements  and  Assessment,  in  1992. 
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o  the  number  of  spare  parts  to  support  the  test 
equipment  would  drop  from  2,000  to  120  items,  saving  about  $9  million. 

Institute  for  Defense  Analyses  Study.  In  May  1993,  the 
Institute  for  Defense  Analyses  issued  a  study  that  estimated  the  costs  and 
benefits  of  Pre-Planned  Product  Improvements  to  the  CASS  Program.  The 
study  stated  that  the  CASS  was  the  most  comprehensive  general-purpose  ATE  in 
DoD.  Congress,  OSD,  and  the  Navy  have  expressed  interest  in  expanding  the 
capability  of  the  CASS  Program  to  test  new,  emerging  avionics  systems  and  in 
applying  CASS  to  Navy  shipboard  electronics  and  other  electronics  for  the 
Army,  the  Air  Force,  and  the  Marine  Corps.  The  study  constructed  and 
analyzed  a  list  of  potential  candidate  Pre-Planned  Product  Improvements  and 
estimated  the  costs  of  the  improvements  by  the  expenses  of  procuring  the 
equipment  and  supporting  software,  integrating  the  system  into  the  CASS 
Program,  and  operating  and  supporting  the  new  capability  at  operational  sites. 
The  benefits  of  the  improvements  were  estimated  by  their  ability  to  test  the 
emerging  avionics  systems  that  are  being  designed  for  new  Navy  and  Air  Force 
aircraft  such  as  the  F/A-18E/F,  F-22,  and  A/F-X.  The  study  was  not  a  cost- 
effectiveness  analysis;  however,  it  did  recommend  a  list  of  Pre-Planned  Product 
Improvements  for  immediate  follow-on  engineering  study.  The  approach 
highlighted  by  the  study  has  significant  merit. 


Effect  of  Milestone  Decision  and  Test  Equipment  Review 


If  the  CASS  Program  does  not  successfully  complete  additional  operational  test 
and  evaluation  and  verify  correction  of  deficiencies  identified  in  testing  and 
configuration  audits,  die  Navy  cannot  be  assured  that  the  CASS  Program  has 
corrected  all  test  deficiencies,  will  not  experience  excessive  rework  and  retrofit, 
and  will  not  require  additional  funding. 

Test  Deficiencies.  By  deferring  many  deficiencies  identified  during 
developmental  and  operational  testing  to  the  Life-Cycle  Test  Program, 
decisionmakers  cannot  be  assured  that  the  CASS  Program  is  ready  for  entry  into 
the  Production  and  Deployment  phase  of  the  acquisition  process.  The  Navy  is 
making  changes  to  the  CASS  Program  to  correct  the  deficiencies  and  is 
producing  CASS  stations  and  subsystems  containing  most  improvements.  Until 
these  CASS  stations  are  tested  further,  however,  decisionmakers  will  not  know 
whether  the  changes  have  resolved  the  previously  identified  problems.  The 
Navy  plans  to  conduct  a  follow-on  operational  test  and  evaluation  in  the  third 
quarter  of  FY  1995  on  CASS  stations  to  verify  that  the  previously  identified 
problems  have  been  resolved;  however,  this  follow-on  operational  test  is 
scheduled  more  than  a  year  after  the  planned  Milestone  HI,  Production 
Approval,  decision  in  March  1994.  Before  the  planned  follow-on  operational 
tests,  the  Navy  will  have  contracted  for  306  CASS  stations,20  which  is  about 

20The  Navy  approved  production  of  55,  60,  and  68  CASS  stations  in  September  1990,  July 
1992,  and  June  1993,  respectively,  and  is  planning  to  approve  production  of  63  and  60  stations 
in  March  1994  and  March  1995,  respectively,  for  a  total  of  306  stations. 
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43  percent  of  the  total  production  of  720  stations.  In  FY  1996,  the  Navy  has 
scheduled  a  Lot  VI  production  buy  for  82  stations.  However,  before  this  buy, 
the  Navy  should  complete  the  OT-m  to  verify  that  deferred  CASS  Program 
deficiencies  have  been  corrected. 

Rework  and  Retrofit.  By  not  ensuring  that  operational  performance 
requirements  are  met  and  functional  and  physical  configuration  audit 
deficiencies  are  corrected  before  establishing  a  production  baseline,  the  Navy 
cannot  be  assured  that  CASS  Program  operational  requirements  are  met  and 
design  deficiencies  have  been  corrected  before  production  that  could  result  in 
excessive  rework  and  retrofit.  In  addition,  the  program  could  experience 
increasing  program  costs  and  a  lengthened  procurement  schedule  as  a  result  of 
not  correcting  deficiencies  identified  in  the  configuration  audits. 

Additional  Funding.  Using  incomplete  test  results  increases  the  risk  of  costly 
retrofits  or  fielding  an  ineffective  system.  Additionally,  the  Program  may 
require  additional  funding  for  life-cycle  testing  associated  with  deferring  the 
correction  of  test  deficiencies. 


Conclusion 


Milestone  Decision.  A  Milestone  m,  Production  Approval,  decision  for  the 
CASS  Program  limited  to  the  Lots  IV  and  V  production  buys  should  be 
conducted  on  the  hybrid,  radio  frequency,  and  CNI  approved  configurations. 
Limiting  the  Milestone  m  decision  to  the  Lots  IV  and  V  production  buys  and 
approved  configurations  would  permit  continued  production.  The  Navy's 
second  Milestone  HI  decision  on  the  electro-optical  configuration  could  then 
review  completion  of  operational  testing  of  corrective  actions  for  deficiencies 
identified  in  prior  operational  tests  that  have  been  deferred  to  the  Life-Cycle 
Test  Program.  Equally  important,  OT-m  test  results  will  provide  operational 
effectiveness  and  suitability  assessment  of  the  CASS  Program  to  accurately 
detect  and  isolate  faults  in  units  undergoing  test.  However,  before  the 
Milestone  m  decision  for  the  Lots  IV  and  V  production  buys,  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  should  review 
the  CASS  Program  cost  and  benefit  analyses  to  determine  whether  the  analyses 
can  be  used  in  lieu  of  a  COEA  to  support  the  Milestone  HI  decision.  The  DoD 
Instruction  5000.2  allows  the  milestone  decision  authority  to  specify  the 
elements  of  a  COEA  for  the  Milestone  m  decision.  However,  a  COEA  of  the 
CASS  Program  may  not  be  needed  to  support  the  Milestone  m  decision  because 
of: 


o  the  cost  benefit  analyses  already  prepared, 
o  the  need  to  proceed  with  the  Program,  and 

o  our  recommendations  in  a  separate  audit  report  to  perform  a  COEA  of 
the  overall  DoD  automatic  test  system  mvestment  strategy. 
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The  award  of  the  Lot  VI  buy  should  be  deferred  until  successful  completion  of 
operational  test  and  evaluation  and  correction  of  the  deficiencies  identified 
during  the  functional  and  physical  configuration  audits.  The  OT-in  phase  of  the 
CASS  Program  should  include  an  evaluation  of  operational  effectiveness  and 
suitability  of  the  CASS  Program.  For  future  Milestone  m  decisions  on  other 
Navy  programs,  policy  should  be  issued  requiring  that  COEAs  be  conducted. 

Cost  and  Operational  Effectiveness  Analysis  Policy.  The  October  9,  1991, 
position  by  the  Office  of  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition)  indicating  that  a  Milestone  HI  decision  does  not 
normally  require  a  COEA  does  not  comply  with  DoD  Instruction  5000.2.  The 
Assistant  Secretary  should  issue  clear  policy  requiring  that  COEAs  be  prepared 
for  all  future  Milestone  IE  decisions  in  accordance  with  DoD  Instruction 
5000.2. 

Director,  Operational  Test  and  Evaluation.  The  Director,  Operational  Test 
and  Evaluation,  should  designate  the  CASS  Program  for  oversight  and  decide 
on  the  operational  effectiveness  and  suitability  of  the  CASS  Program  in  support 
of  the  recommended  Navy  Program  Decision  Meeting  before  the  Lot  VI 
production  buy.  To  verify  that  the  CASS  Program  is  operationally  ready  for 
production  and  deployment  and  suitable  for  fulfilling  a  variety  of  DoD  ATE 
requirements,  the  CASS  Program  should  be  subject  to  test  and  evaluation 
oversight  by  the  Director,  Operational  Test  and  Evaluation.  Copies  of  the 
formal,  detailed  developmental  and  operational  test  and  evaluation  reports  of  the 
results,  conclusions,  and  recommendations  prepared  at  the  end  of  each  phase  of 
developmental,  operational  test  and  evaluation,  and  field  reliability  reports 
should  be  provided  to  the  Director,  Operational  Test  and  Evaluation,  for  an 
assessment  of  test  adequacy  and  the  Director's  assessment  of  the  CASS 
Program's  operational  effectiveness  and  suitability.  This  assessment  is 
particularly  important  since  the  CASS  Program  may  be  considered  as  a  standard 
ATE  set  for  use  by  other  Military  Departments.  The  decision  concerning  the 
operational  effectiveness  and  suitability  of  the  CASS  Program  should  be 
provided  to  the  Assistant  Secretary  of  the  Navy  (Research,  Development  and 
Acquisition)  for  consideration  at  the  recommended  Navy  Program  Decision 
Meeting. 


Recommendations  for  Corrective  Action 


1.  We  recommend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition): 

a.  Conduct  a  Milestone  HI,  Production  Approval,  decision  for  the 
Consolidated  Automated  Support  System  Program  limited  to  the  Lots  IV 
anH  V  production  buys  and  the  hybrid,  radio  frequency,  and 
Communications/  Navigation/Identification  configurations. 

b.  Defer  the  award  of  the  Lot  VI  production  buy  decision  until  the 
results  of  operational  test  and  evaluation  through  the  Operational 
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Testing-m  phase  are  available  and  a  Navy  program  decision  meeting  to 
review  the  overall  Consolidated  Automated  Support  System  Program  and, 
in  particular,  the  electro-optical  configuration  is  completed. 

c.  Require  the  Operational  Testing-IH  phase  include  operational 
effectiveness  and  suitability  of  the  Consolidated  Automated  Support  System 
Program  to  accurately  detect  and  isolate  faults  within  Units  Under  Test 
when  executing  maintenance  and  production  weapons  systems  test  program 
sets,  to  verify  correction  of  deficiencies  identified  in  Operational 
Testing-HC,  and  to  complete  deferred  or  incomplete  operational  test  and 
evaluation. 

d.  Require  correction  of  the  deficiencies  identified  during  functional 
and  physical  configuration  audits  of  the  Consolidated  Automated  Support 
System  Program.  Correction  is  defined  as  successful  completion  of  a  plan 
of  action  by  the  contractor  with  mutual  agreement  by  the  Navy  on  the 
corrective  actions  to  be  incorporated  into  production  units  before  the  Lot  V 
production  decision. 

e.  Decide  whether  the  Consolidated  Automated  Support  System 
Program  cost  and  benefit  analyses  contain  the  elements  of  a  cost  and 
operational  effectiveness  analysis  needed  to  support  the  Milestone  HI, 
Production  Approval,  decision. 

f.  Issue  clear  policy  for  future  program  decisions  that  require  cost 
and  operational  effectiveness  analyses  be  conducted  for  Milestone  m,  in 
accordance  with  DoD  Instruction  5000.2,  "Defense  Acquisition  Policies  and 
Procedures,"  February  23,  1991,  and  DoD  Manual  5000.2-M,  "Defense 
Acquisition  Management  Documentation  and  Reports,"  February  23, 1991. 

2.  We  recommend  that  the  Director,  Operational  Test  and  Evaluation: 

a.  Designate  the  Consolidated  Automated  Support  System  Program 
for  program  oversight,  including  review  of  the  test  and  evaluation  master 
plan. 


b.  Render  a  decision  concerning  the  operational  effectiveness  and 
suitability  of  the  Consolidated  Automated  Support  System  Program  based 
on  the  results  of  Operational  Testing-m  in  support  of  the  recommended 
Navy  program  decision  meeting  before  the  Lot  VI  production  buy. 


Management  Comments  and  Audit  Response 


Management  Comments.  We  did  not  receive  formal  comments  from  the 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  and 
the  Director,  Operational  Test  and  Evaluation,  to  a  draft  of  this  report  issued 
March  17,  1994.  The  comments  were  required  by  May  16,  1994. 
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Audit  Response.  The  DoD  Directive  7650.3  requires  that  all  audit 
recommendations  be  resolved  promptly.  Although  we  understand  that  the  Navy 
has  taken  action  to  address  the  concerns  raised  in  this  report  and  to  implement 
the  recommendations,  confirmation  is  needed,  as  well  as  estimated  dates  of 
completion  for  all  actions.  Therefore,  we  request  that  the  Assistant  Secretary 
and  the  Director  provide  comments  on  the  final  report. 
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Appendix  A.  Consolidated  Automated  Support 

System  Station  Configurations 


Basic  Test  Capability 
plus: 

♦  Communications 

•  Navigation 

*  Spread  Spectrum 
Systems 


Basic  Test  Capability 
plus: 

•  Electronic  Counter- 
Measures 

•  Electronic  Counter- 
Counter-Measures 

•  Electronic  Warfare 
Support  Measures 

•  Fire  Control  Radar 

•  Navigation  Radar 

•  Tracking  Radar 

•  Surveillance  Radar 

•  Radar  Altimeter 


Hybrid  Station 

(Basic  Teat  Station) 


•  General  Purpose 
Electrical  /  Electronics 

•  Computers 

•  Instruments 

•  Flight  Controls 

•  Plus  Subsystems  for 

•  Pneumatic 

•  Display 

•  Inertial  Navigation 


EO  Station 


Basic  Test  Capability 
plus' 

•  Forward  Looking 
Infrared  (FLIR) 

•  Lasers  /  Designators 

•  Laser  Range  Finders 

•  Visual  TV  Systems 


Appendix  B.  Changes  in  Consolidated 

Automated  Support  System  Test 
Characteristics 


The  CASS  test  characteristics  in  the  Test  and  Evaluation  Master  Plans  used  for 
the  OT-HB  and  DT-IIC-3  were  compared  for  differences  in  technical  and 
operational  characteristics  and  thresholds.  The  Test  and  Evaluation  Master  Plan 
No.  778,  revision  1,  change  1,  January  3,  1992,  approved  by  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition),  was  used  to 
conduct  OT-IIB  testing.  The  Test  and  Evaluation  Master  Plan  No.  778, 
revision  1,  change  2,  May  16,  1993,  approved  by  the  Acting  Assistant  Secretary 
of  the  Navy  (Research,  Development  and  Acquisition),  was  used  to  conduct 
DT-HC-3  and  OT-HC  testing. 

Critical  Technical  Characteristics.  The  following  critical  technical 
characteristics  were  in  change  1  but  not  in  change  2: 

o  Mean-Time-To-Calibrate 

o  Maximum  Annual  Scheduled  Maintenance 

o  Time  Constraints  for  Fault  Isolation 

Technical  Threshold.  One  technical  threshold  in  change  1  was  reduced  in 
change  2:  Test  Program  Set  Transferability  was  reduced  from  100  percent  to 
90  percent. 

Operational  Characteristic  Threshold.  One  operational  characteristic 
threshold  in  change  1  was  excluded  in  change  2:  the  time  constraints  for  Fault 
Isolation. 

Suitability  Parameter.  The  following  suitability  parameters  in  change  1  were 
excluded  in  change  2: 

o  Mean-Time-To-Calibrate 

o  Maximum  Annual  Scheduled  Maintenance 

o  Maximum  Corrective  Maintenance  Time 
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Resulting  From  Audit 


Recommendation 

Reference 

Description  of  Benefit 

Amount  and/or 

Type  of  Benefit 

l.a. 

Program  Results.  Will  validate  the 
production  of  the  CASS  Program 
limited  to  the  Lots  IV  and  V 
production  buys. 

Benefits  not 
quantifiable  because 
file  benefits  will 
depend  on  future 
actions  by  the  Navy. 

l.b. 

Program  Results.  Will  verify  the 
deferral  of  the  Lot  VI  production 
buy  decision  until  operational  test 
and  evaluation  is  successfully 
completed. 

Benefits  not 
quantifiable  because 
the  benefits  will 
depend  on  future 
actions  by  the  Navy. 

l.c. 

Program  Results.  Will  verify  that 
the  Operational  Testing-III  phase 
includes  operational  effectiveness 
and  suitability. 

Benefits  not 
quantifiable  because 
the  benefits  will 
depend  on  future 
actions  by  the  Navy. 

l.d. 

Program  Results.  Will  verify 
correction  of  the  deficiencies 
identified  during  functional  and 
physical  configuration  audits  of  the 
CASS  Program. 

Benefits  not 
quantifiable  because 
tiie  benefits  will 
depend  on  future 
actions  by  the  Navy. 

l.e. 

Program  Results.  Will  verify  that 
CASS  Program  cost  and  benefit 
analyses  contain  the  elements  of  a 
cost  and  operational  effectiveness 
analysis. 

Benefits  not 
quantifiable  because 
tiie  benefits  will 
depend  on  future 
actions  by  the  Navy. 

l.f. 

Program  Results  and  Internal 

Controls.  Will  validate  that  cost 
and  operational  effectiveness 
analyses  are  prepared  for  future 
Milestone  HI  decisions. 

Benefits  not 
quantifiable  because 
the  benefits  will 
depend  on  future 
actions  by  the  Navy. 

2. 

Program  Results.  Will  provide  an 
independent  decision  as  to  the 
operational  effectiveness  and 
suitability  of  the  CASS  Program. 

Benefits  not 
quantifiable  because 
the  benefits  will 
depend  on  future 
actions  by  OSD. 
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Appendix  D.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington,  DC 
Assistant  Secretary  of  Defense  (Economic  Security),  Washington,  DC 
Director,  Acquisition  Program  Integration,  Washington,  DC 
Director,  Tactical  Systems,  Washington,  DC 
Comptroller  of  the  Department  of  Defense,  Washington,  DC 
Director,  Operational  Test  and  Evaluation,  Washington,  DC 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition), 
Washington,  DC 

Naval  Air  Systems  Command,  Arlington,  VA 

Consolidated  Automated  Support  System  Program  Office,  Arlington,  VA 
Naval  Air  Technical  Services  Facility,  Washington,  DC 
Naval  Air  Warfare  Center,  Aircraft  Division,  Lakehurst,  NJ 
Naval  Air  Warfare  Center,  Aircraft  Division,  Patuxent  River,  MD 
Naval  Sea  Systems  Command,  Arlington,  VA 
Space  and  Naval  Warfare  Systems  Command,  Arlington,  VA 
Commander,  Operational  Test  and  Evaluation  Force,  Norfolk,  VA 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition),  Washington,  DC 

Deputy  Assistant  Secretary  (Communications,  Computer  and  Support  Systems), 
Washington,  DC 


Defense  Agencies 

Defense  Contract  Audit  Agency,  Alexandria,  VA 
Defense  Logistics  Agency,  Alexandria,  VA 

Defense  Contract  Management  Command,  Alexandria,  VA 
Defense  Contract  Management  Area  Office,  Orlando,  FL 


Contractor 

Martin  Marietta,  Daytona  Beach,  FL 
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Department  of  the  Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Air  Force  Audit  Agency 


Defense  Agencies 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Logistics  Agency 

Commander,  Defense  Contract  Management  Command 
Defense  Contract  Management  Area  Office,  Orlando,  FL 
Inspector  General,  Defense  Intelligence  Agency 
Inspector  General,  National  Security  Agency 
Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Organizations 

Office  of  Management  and  Budget  .  . 

U.S.  General  Accounting  Office,  National  Security  and  International  Affairs  Division, 
Technical  Information  Center 

Chairman  and  Ranking  Minority  Member  of  the  Following  Congressional  Committees 
and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on 
Government  Operations 
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Audit  Team  Members 


Donald  E.  Reed 

Russell  A.  Rau 

Jack  D.  Snider 
Cordelia  Grace-Scott 
Debbie  A.  Calhoun 
Mary  Ann  Hourcl6 
Teresa  D.  Bone 


Director,  Acquisition  Management 
Directorate 

Audit  Program  Director,  Systems 
Acquisition  Division 
Audit  Project  Manager 
Senior  Auditor 
Auditor 
Editor 

Administrative  Support 
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